1. Field of the Invention
The present invention relates to a method and system for haptic interaction in augmented reality. More specifically, the present invention relates to a method and system for haptic interaction in augmented reality that can effectively remove noise from real images captured by an image capturing device and minimize discontinuity of force generated in the haptic interaction so as to stably and smoothly perform the haptic interaction in the augmented reality.
2. Description of the Related Art
Generally, augmented reality (AR) means a technology that helps to display virtual objects as if the virtual objects exist in the same space as a real environment. In other words, the augmented reality is a virtual reality that displays the real world seen through eyes of users and the virtual world with one image having additional information by binding them. A study on the augmented reality has been frequently performed since users interacting within a virtual environment produced by a computer are limited in feeling reality approximately similar to the real environment.
The general AR system mixes images captured from cameras with the virtual objects registered in the virtual environment of the users. Various AR technologies for providing the interaction or the display, which immerses the users in various application fields, such as medical services, military affairs, industries, educations, entertainments, and the like, have been suggested. The AR technology can be applied to all the sense organs, such as sense of sight, auditory sense, tactile sense, sense of taste, sense of smell, and the like; however, in reality, most AR systems are concentrated on the sense of sight.
On the other hand, a haptic technology can further immerse to the users in AR regions and perform the interactive operations along with the sense of feel. The haptic technology is to provide a variety of information on the virtual or real environment to the users through kinesthetic and tactile. The word “haptic” means tactile sense in Greek and is a concept including both kinesthetic and tactile. Unlike other senses, the haptic has an interactive feature that can perform the operation along with the feeling. Therefore, in order to increase the immersion for users of a vision-based augmented reality system, an attempt to unite an acoustic effect and a haptic device 150 has been recently made.
FIG. 1 is a diagram showing one example of a vision-based augmented reality system using an AR marker and a haptic device.
As shown in FIG. 1, a vision-based augmented reality system 100 includes AR markers 110, an image capturing device 120, a main processing device 130, a visual display device 140, and a haptic device 150.
The AR markers 110 can be displayed in predetermined patterns. One AR marker 110 is established to correspond to one virtual object. The markers 110 used in the virtual reality may be two-dimensional or three-dimensional images in addition to the AR markers 110 having the predetermined patterns.
The image capturing device 120 performs a function of capturing images for a real environment in addition to the AR markers 110. As a representative example, there may be a camera.
The main processing device processes images input from the image capturing device 120, recognizes the AR markers 110 among the input images, and produces virtual objects corresponding to the AR markers 110.
The visual display device 140 visually outputs images that are processed in the main processing device. At this time, the visual display device 140 outputs the images of the virtual objects corresponding to the AR markers 110 as two dimensional or three-dimensional images together with the images for the real environment.
The haptic device 150 performs haptic interaction with the virtual objects of the virtual environment. At this time, the haptic device provides reaction force according to the collision with the virtual objects to users and enables the users to operate the virtual objects.
The operation of the vision-based augmented reality system will be described. If the image capturing device 120 receives the images for the real environment in addition to the AR markers 110, the main processing device outputs the input images to the visual display device 140. At this time, the main processing device produces virtual objects corresponding to the AR markers 110 and output the real images and the virtual objects to the visual display device 140. Meanwhile, the main processing device detects whether end points (or haptic interaction points) of the haptic device 150 collide with the virtual objects and when the end points collide with the virtual objects, calculates reaction force and transfers the signal of the calculated reaction force to the haptic device 150, such that the users can feel the reaction force.
The vision-based augmented reality method augments virtual objects in live images captured from the image capturing device. However, there is a problem in that the virtual objects virtually tremble due to a tracking noise of the marker that is a reference. Further, when the users perform haptic interaction with the virtual objects, the noise can cause a trembling of force or unstable haptic interaction. Also, since the coordinates of the image capturing device are different from the coordinates of the haptic environment, in order to provide realistic haptic interaction, each coordinate system should be united. Further, when the virtual objects registered on the marker, which are reference points, slightly move rapidly, the estimated moving distance may be discontinuous between frames. This disparity can be tolerable in terms of the visual aspect, but may be very important problem to solve in terms of haptic interaction. The information can be sufficiently and stably transferred when the frame is refreshed for about every 30 Hz in terms of the image rendering, but stable haptic interaction can be performed only when the frame is refreshed for each 1 kHz in terms of haptic rendering.
Therefore, a need exists for a method that can minimize noise in the marker tracking, conform the coordinates of the images to the coordinates of the haptic, and achieve smooth haptic interaction.